{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Enviroment: \n",
    "Open AI gym [MountainCar-v0](https://github.com/openai/gym/wiki/MountainCar-v0)\n",
    "\n",
    "### Observation\n",
    "Type: Box(2)\n",
    "\n",
    "| Num  | Observation | Min   | Max  |\n",
    "| ---- | ----------- | ----- | ---- |\n",
    "| 0    | position    | -1.2  | 0.6  |\n",
    "| 1    | velocity    | -0.07 | 0.07 |\n",
    "\n",
    "### Actions\n",
    "\n",
    "Type: Discrete(3)\n",
    "\n",
    "| Num  | Action     |\n",
    "| ---- | ---------- |\n",
    "| 0    | push left  |\n",
    "| 1    | no push    |\n",
    "| 2    | push right |\n",
    "\n",
    "### Reward\n",
    "\n",
    "-1 for each time step, until the goal position of 0.5 is reached. As with MountainCarContinuous v0, there is no penalty for climbing the left hill, which upon reached acts as a wall.\n",
    "\n",
    "### Starting State\n",
    "\n",
    "Random position from -0.6 to -0.4 with no velocity.\n",
    "\n",
    "### Episode Termination\n",
    "\n",
    "The episode ends when you reach 0.5 position, or if 200 iterations are reached.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. gym enviroment setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.44864847,  0.        ])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import gym\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "env = gym.make(\"MountainCar-v0\")\n",
    "env.reset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Q Table setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "LEARNING_RATE = 0.5\n",
    "DISCOUNT = 0.95\n",
    "EPISODES = 10000\n",
    "SHOW_EVERY = 200\n",
    "Q_TABLE_LEN = 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "DISCRETE_OS_SIZE = [Q_TABLE_LEN] * len(env.observation_space.high)\n",
    "discrete_os_win_size = (env.observation_space.high - env.observation_space.low) / DISCRETE_OS_SIZE\n",
    "\n",
    "# q_table = np.random.uniform(low=0, high=1,\n",
    "#                             size=(DISCRETE_OS_SIZE + [env.action_space.n]))\n",
    "\n",
    "q_table = np.zeros(DISCRETE_OS_SIZE + [env.action_space.n])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(20, 20, 3)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "q_table.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decay epsilon "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "epsilon = 1  # not a constant, qoing to be decayed\n",
    "START_EPSILON_DECAYING = 1\n",
    "END_EPSILON_DECAYING = EPISODES//2\n",
    "epsilon_decay_value = epsilon/(END_EPSILON_DECAYING - START_EPSILON_DECAYING)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Help functions "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_discrete_state (state):\n",
    "    discrete_state = (state - env.observation_space.low) // discrete_os_win_size\n",
    "    return tuple(discrete_state.astype(int))\n",
    "\n",
    "def take_epilon_greedy_action(state, epsilon):\n",
    "    discrete_state = get_discrete_state(state)\n",
    "    if np.random.random() < epsilon:\n",
    "        action = np.random.randint(0,env.action_space.n)\n",
    "    else:\n",
    "        action = np.argmax(q_table[discrete_state])\n",
    "    return action"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Rewards Recorder setup "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "ep_rewards = []\n",
    "aggr_ep_rewards = {'ep':[],'avg':[],'min':[],'max':[]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Train the Agent "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "episode: 0\n",
      "episode: 200\n",
      "episode: 400\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-38-c828d9c97a74>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     22\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mdone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     23\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 24\u001b[0;31m             \u001b[0mtd_target\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mreward\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mDISCOUNT\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmax\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mq_table\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mget_discrete_state\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnext_state\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     25\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     26\u001b[0m             \u001b[0mq_table\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mget_discrete_state\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstate\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0maction\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0mLEARNING_RATE\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mtd_target\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mq_table\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mget_discrete_state\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstate\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0maction\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "for episode in range(EPISODES):\n",
    "    # initiate reward every episode\n",
    "    ep_reward = 0\n",
    "    if episode % SHOW_EVERY == 0:\n",
    "        print(\"episode: {}\".format(episode))\n",
    "        render = True\n",
    "    else:\n",
    "        render = False\n",
    "\n",
    "    state = env.reset()\n",
    "    done = False\n",
    "    while not done:\n",
    "        action = take_epilon_greedy_action(state, epsilon)\n",
    "\n",
    "        next_state, reward, done, _ = env.step(action)\n",
    "\n",
    "        ep_reward += reward\n",
    "\n",
    "        # if render:\n",
    "        #     env.render()\n",
    "\n",
    "        if not done:\n",
    "\n",
    "            td_target = reward + DISCOUNT * np.max(q_table[get_discrete_state(next_state)])\n",
    "\n",
    "            q_table[get_discrete_state(state)][action] += LEARNING_RATE * (td_target - q_table[get_discrete_state(state)][action])\n",
    "\n",
    "        elif next_state[0] >= 0.5:\n",
    "            # print(\"I made it on episode: {} Reward: {}\".format(episode,reward))\n",
    "            q_table[get_discrete_state(state)][action] = 0\n",
    "\n",
    "\n",
    "        state = next_state\n",
    "\n",
    "    # Decaying is being done every episode if episode number is within decaying range\n",
    "    if END_EPSILON_DECAYING >= episode >= START_EPSILON_DECAYING:\n",
    "        epsilon -= epsilon_decay_value\n",
    "\n",
    "    # recoard aggrated rewards on each epsoide\n",
    "    ep_rewards.append(ep_reward)\n",
    "\n",
    "    # every SHOW_EVERY calculate average rewords\n",
    "    if episode % SHOW_EVERY == 0:\n",
    "        avg_reward = sum(ep_rewards[-SHOW_EVERY:]) / len(ep_rewards[-SHOW_EVERY:])\n",
    "        aggr_ep_rewards['ep'].append(episode)\n",
    "        aggr_ep_rewards['avg'].append(avg_reward)\n",
    "        aggr_ep_rewards['min'].append(min(ep_rewards[-SHOW_EVERY:]))\n",
    "        aggr_ep_rewards['max'].append(max(ep_rewards[-SHOW_EVERY:]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'Rewards')"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZQAAAEKCAYAAAA1qaOTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl4lOXV+PHvyb6HJAQIBEjCjmERArJYxbqBVgHFrbZarWJrfdX251JrbevbfVOr9bXSVlGrokJRFBXFDTdWBQKEQIAgISzZ933u3x/zJA7JJJkkM5mZ5Hyuay4mz3o/mZCTezu3GGNQSimleirA2wVQSinVN2hAUUop5RYaUJRSSrmFBhSllFJuoQFFKaWUW2hAUUop5RYaUJRSSrmFBhSllFJuoQFFKaWUWwR5uwCeNnDgQJOSkuLtYiillN/Ytm1boTEmsavn9fmAkpKSwtatW71dDKWU8hsicrg752mTl1JKKbfQgKKUUsotNKAopZRyiz7fh+JMQ0MDeXl51NbWersoHhEWFkZycjLBwcHeLopSqh/plwElLy+P6OhoUlJSEBFvF8etjDEUFRWRl5dHamqqt4ujlOpH+mWTV21tLQkJCX0umACICAkJCX229qWU8l39MqAAfTKYNOvLz6aU8l39sslLKdV/GWPYfHwzW09sxdkS6GFBYZw/8nxGxozs9FpVDVW8k/sOsaGxnJV8FkEBHf9KNcbw5ckv2XhsIzZjc3rvK8ZeQWxorOsP5EM0oCil+oXK+krWHFjDS9kvcbDsIABC29q8wfC3L/7G3KFzuXr81Xxj2DcIDAg85ZiDpQdZkb2CNQfWUNVQBcCQyCFcOfZKLhtzGQnhCaccX91QzdpDa1mxdwX7SvZ1eO9Xc17l79/8OymxKe547F4lziJ0X5KRkWFaz5TPyspiwoQJXipR7+gPz6iUKw6UHuDFvS/y+oHXqW6sJj0hnavHX8381PmEBoa2Ob6wppCV+1bySvYrnKw5ybCoYVwx9goWjl7IjpM7eHHvi2w6vonggGAuTLmQq8ZdRVFNES9mv8imY/btF6RcwNXjriYuLI4Ve1fwWs5rVDRUMC5uHFePv5qLUi8iIjiizb2/PPkld35wJw22Bh6a9xCzkmZ1+7nrm+oJCQzp1rkiss0Yk9Hl8zSgeMeiRYs4cuQItbW13HHHHTQ1NXHo0CH+9Kc/AbB8+XK2bdvGY489xq9//Wuef/55hg8fzsCBA5k+fTp33XVXh9f3hWdU/cv/vP8/XJJ2CRekXNCj62QVZfHg5w/y93P/zsDwgT261hM7nuD/tv8fIQEhzE+dzzXjryF9YLpL5zbYGvjgqw9Ykb2CLce3tGxPikziynFXsnj04jY1kYNlB3lp70usObCGyoZKAIIkiPNHns81E65hauLUTvs4j1Ye5bb3buNQ2SF+dsbPuHLclS4/r83Y+Cz/M1bsXcHxquO8cskr3epT7W5A6fdNXg++vps9+eVuvebEoTH88pLTOjzmqaeeIj4+npqaGmbMmMF7773H3LlzWwLKSy+9xP3338/WrVtZtWoVX375JY2NjUybNo3p06e7tbxK9VRNYw0fHvmQhqaGHgeUNw+9ye6i3azev5qbJ9/co2u9fuB1pg2axiPnPEJcWFyXzm2uaVyQcgE5JTmsO7yOCfETOuwrSYtN474z7uOOaXew9tBaKusruWTUJV0KjMOihvHcgue4Z8M9/HrjrzlYdpC7Mu7qsH+mrK6M13Je46Xsl/iq4iviw+JZMnYJDbaGbtdSuqPfBxRvefTRR1m9ejUAR44c4dChQ6SlpbFx40bGjBlDdnY2c+fO5W9/+xsLFy4kPDwcgEsuucSbxVbKqZLaEgC2ndjWo6YWgI3HNgKwav8qvj/p+wRI9wajlteXc6TiCItHL+5yMGltdNxoRseNdvn4iOAIrhh7RbfvFxUSxWPffIyHtj3Es3ueJbcslyvHXdmm38VmbHx89GPWHlxLbVMtUxOn8qOpP+L8kecTHNj7E5v7fUDprCbhCR9++CHr16/n888/JyIignnz5lFbW8tVV13Fyy+/zPjx41m8eDEi4nQUilK+pri2GIDaplp2FOxgxpAZ3b7O3uK9TIifQFZxFhuPbWTO0DndulZWURYAExMmdut8bwsMCOTuGXeTFpvGbzb+hk/zP3V6XFhgGBenXczV469mfPz4Xi7lqfp9QPGGsrIy4uLiiIiIYO/evWzcaP+L7LLLLuO3v/0tI0eO5I9//CMAZ555Jrfccgv33XcfjY2NrF27lptv7lkzgFLu1hxQAD7P/7zbAWXzsc0A3DvzXu784E5W7lvZ7YCyp2gP4L8BpdnlYy/nzGFnUlRb5HR/cnQyMSExvVwq5zSgeMH8+fP5xz/+weTJkxk3bhyzZtlHcsTFxTFx4kT27NnDzJkzAZgxYwaXXnopU6ZMYeTIkWRkZBAb659j1FXfVVRj/2U3KGIQm45t6vZ1Nh7bSHRwNFMTp7Jw1EKez3qewprCbnXOZxVlkRSZ1OPmLl8wOHIwgyMHe7sYneq3M+W9KTQ0lLfeeoudO3fyyiuv8OGHHzJv3jwA3njjDQ4ePHjK8XfddRfZ2dm8+uqrZGdna6e88jkldfY+lAUpC9hVtIvy+q4PdDHG8Hn+58xMmklgQCCXj72cRtPIazmvdatMe4r3+H3txN9oQPEDS5cuZerUqUybNo3LL7+cadOmebtISp2iuKaY8KBw5g2fh83YThlm66q8ijzyq/I5I+kMAFJjU5k+eDqr9q9yOqu8IxX1FRwuP6wBpZdpk5cfeOGFF7xdBKU6VFxbTHxYPFMSpxAeFM7G/I2cO+LcLl3j82OfA5wymW/J2CXc9/F9bD6+uUuT/PYW7wX8v//E32gNRSnVY80BJTgwmOmDp7cM/e2Kjcc2MjhiMCkxKS3bzh95PjEhMazct7JL12rukJ8Qr5N7e5NXAoqIXCEiu0XEJiIZDttnish267VDRBY77JsvItkikiMiP/VGuZVSzjUHFLDXMHLLczleddzl85tsTS21EMeZ3aGBoVw66lLe++q9lo5/V+wu2s3giMFtZrIrz/JWDWUXcBmwwcn2DGPMVGA+8KSIBIlIIPA4sACYCFwjIlqXVcpHFNUWnRJQgC7VUvaW7KWsroxZQ9s2ay0Zu4RGWyNrDqxx+XpZRVna3OUFXgkoxpgsY0y2k+3VxphG68swoHlW30wgxxhz0BhTD6wAFvZOaZVSHTHGUFJb0hJQxsSNIT4svksBZWO+/Vhn/SSjBoxi2qBprNq/yqWJvpX1leSW52pA8QKf60MRkTNEZDeQCfzACjDDgCMOh+VZ2/qsNWvW8Ic//MHbxVCqU5UNlTTYGloCSoAEcEbSGWzM3+hypoeNxzYyesDoduebLBm7hMPlh9l6YqvT/Y6yiv17hrw/81hAEZH1IrLLyavDmoUxZpMx5jRgBnCfiISBk4UDvq69OLv3UhHZKiJbCwoKevYgXnLppZfy059qV5Hyfc2z5B0nEM5Omk1RbRE5pTmdnl/bWMsXJ77ocBTX+SPPJzokmlf2vdLp9frKDHl/5LGAYow5zxiT7uTl0iwlY0wWUAWkY6+RDHfYnQzkd3DuMmNMhjEmIzExsSeP4RG5ubmMHz+em266ifT0dK699lrWr1/P3LlzGTNmDJs3b2b58uXcdtttAHzve9/j9ttvZ86cOaSlpbFyZddGvCjlSc0BJSHs6w7wrvSjbC/YTr2tntlDZ7d7TFhQGJekXcL6w+tbElG2Z0/RHgZFDOpx6nvVdT41D0VEUoEjxphGERkJjANygVJgjLX/KHA18G233PStn8LxTLdcqsWQSbCg4+aqnJwcXnnlFZYtW8aMGTN44YUX+OSTT1izZg2/+93vWLRo0SnHHzt2jE8++YS9e/dy6aWXsmTJEveWWaluKq6xB5T48PiWbUlRSYyMGcnGYxv57sTvdnj+xvyNBEkQ0wd3nAFiydglvLD3BdYcWMP1p13f7nF7inSGvLd4a9jwYhHJA2YDa0VknbXrTGCHiGwHVgO3GmMKrX6U24B1QBbwsjFmtzfK7i6pqalMmjSJgIAATjvtNM4991xEhEmTJpGbm9vm+EWLFhEQEMDEiRM5ceJE7xdYqXY0Jy1s7kNpNitpFluPb6XB1tDh+ZuObWJy4mQigyM7PG5M3BimJE5h5b6V7fbNVDVU6Qx5L/JKDcUYsxp7wGi9/TnguXbOeRN40+2F6aQm4SmhoV8vPRoQENDydUBAAI2NjR0eryntlS9pboKKCz01CeOspFm8lP0Suwp3cfqg052eW1ZXxu6i3fxwyg9duteSsUt44NMH2HZiGxlD2i4omFWUhcFwWkLvL0uhfHCUl1LKvxTXFhMTEtNmQacZQ2YgSMuQYGe2HN+CwTidf+LMhSkXEh0czcr9zvsRtUPeuzSgKKV6xHGWvKPY0FgmJkzssGN+47GNRARFuLzOe3hQOBenXcy7ue9SVlfWZv+e4j0MCtcOeW/xqU75/iIlJYVdu3a1fL18+XKn+773ve+12Q9QWVnp6SIq5bL2AgrYm72e2f0MVQ1VTvtINh7bSMaQDIIDXF+udsnYJazIXsHrB17nOxO/c8o+nSHvXVpDUUr1SIcBZegsGk0j205sa7MvvzKfw+WHu5RFGGBc/DgmD5zcpnO+uqGaQ2WHNKB4kdZQlFI9UlxbzLRBztfoOX3Q6YQGhrJ893J2F546MPNgmX0hua4GFLDXUn7x2S/YXrC9pcN/b/FeDEYDihdpQFFKdVuTrYnSutJT5qA4Cg0M5byR57H24Fqni26NjRvL6AGju3zfC1Mu5I9b/sjKfStbAop2yHufBhSlVLeV1ZdhM7Y2Q4Yd/f7M3/P7M3/f7n7HdPWuigiO4OLUi3ntwGvcM+MeYkNj2VO0h8TwRBIjfC87Rn+hfShKqW5zNku+NRHp8NVdS8Yuoa6pjjcOvgHoDHlfoAFFKdVtzvJ49ZYJCRM4LeE0Vu5bae+QLz/EhARdodGbNKAopbqtOaC0N8rL05aMXUJOaQ4vZ7+MzdiYGK81FG/SgKKU6rb28nj1lgWpC4gIiuCJHU8A2iHvbRpQvMCV9PWbN29mzpw5nH766cyZM4fsbPsClw899BA33ngjAJmZmaSnp1NdXe3Nx1H9WEltCQESQGxorFfuHxkcyUVpF1HdWE1CWAKDIgZ5pRzKrt+P8vrj5j+yt3ivW685Pn489868t8NjOktf/+yzz7JhwwaCgoJYv349P/vZz1i1ahV33nkn8+bNY/Xq1fz2t7/lySefJCIiwq3lV8pVxbXFDAgdQIB472/TJWOXsHLfSiYmTOxRJ7/quX4fULylOX094DR9fVlZGddffz379+9HRGhosKcADwgIYPny5UyePJlbbrmFuXPnevMxVD/X0Sz53nJawmlcNe4qZie1v0CX6h39PqB0VpPwlM7S1z/wwAOcc845rF69mtzcXObNm9dy/P79+4mKiiI/v91FK1U/Vd1QTXVj15pAY0Ni22QKdlVxbbFXRni19vNZP/d2ERQaUHxWWVkZw4YNA05NDllWVsYdd9zBhg0buO2221i5cqWu3qgA+9oiC/67gIr6ii6dlxyVzH8u+g8J4V0PDMW1xTqySrXQgOKj7rnnHq6//noeeughvvnNb7Zs//GPf8ytt97K2LFj+fe//80555zDWWedxaBB2hnZ371x8A0q6iu4Y9odxITEuHRObWMtj375KHd9dBfLLljWpay/YJ/Y2NGkRtXPGGN6/QVcAewGbECGk/0jgErgLodt84FsIAf4qav3mj59umltz549bbb1Nf3hGdXXbDabWfTqInPNG9d0+dw1OWtM+vJ084dNf+jSefWN9SZ9ebp5cseTXb6n8m3AVtON3+3eGpqxC7gM2NDO/oeBt5q/EJFA4HFgATARuEZEtJ6tlGVHwQ5ySnNYMrbrzZ+XjLqE70z4Dv/J+g+vH3jd5fOaJzXGhbWfx0v1L14JKMaYLGNMtrN9IrIIOIi9BtNsJpBjjDlojKkHVgALPV9SpfzDyn0riQyOZH7K/G6d/5OMnzBjyAwe/PzBlqy9nfH2LHnle3xqYqOIRAL3Ag+22jUMOOLwdZ61rduMw8I8fU1ffjbVVnl9Oety13FR6kVEBHdvTlJwQDB/PuvPDAgdwJ0f3ElJbUmn53gzj5fyTR4LKCKyXkR2OXl1VLN4EHjYGNN6jVtns5Xa/a0pIktFZKuIbC0oKGizPywsjKKioj75i9cYQ1FREWFhYd4uiuolaw+upbaptlvNXY4SwhN45JxHKKop4u4Nd9Noa+zweK2hqNY8NsrLGHNeN047A1giIn8CBgA2EakFtgHDHY5LBtqdhGGMWQYsA8jIyGgTNZKTk8nLy8NZsOkLwsLCSE5O9nYxVC8wxvDKvleYmDDRLXms0gem88DsB3jg0wd4ZNsj3DXjrnaP1YCiWvOpYcPGmG80vxeRXwGVxpi/i0gQMEZEUoGjwNXAt7t7n+DgYFJTU3taXKW8LrMwk/0l+/nF7F+47ZqLRi9iZ8FOntnzDNdOuJakqCSnxxXXFhMSEEJkcKTb7q38m1f6UERksYjkAbOBtSKyrqPjjTGNwG3AOiALeNkYs7ujc5TqD1buW0l4UDgXpV7k1usuGr0IgN1F7f83K64tJi4sTvNnqRZeqaEYY1YDqzs55letvn4TeNODxVLKr1TUV/B27ttclHqR22sJY+PGEiiB7Cnaw3kjnbde+0IeL+VbfGqUl1LKdW8efJOaxhquGHuF268dFhTGqAGjOhxCrLPkVWsaUJTyQ8YYVu5fyYT4CR5bVGpiwkT2FO1pdzSkrySGVL5DA4pSfmhP0R72Fu/l8jGXe6wPY2LCRErqSjheddzpfm3yUq1pQFHKD72y7xV7Z3yaezvjHTXXfJw1e1U3VFPbVKsBRZ1CA4pSfqaqoYo3D73J/JT5RIdEe+w+4+LGESiBTkd6aR4v5YwGFKX8zPrD66lprOGyMZd59D5hQWGkDUhjT3HbGopOalTOaEBRys+8lfsWw6KGMSVxisfvNTF+IllFWW065jWPl3JGA4pSfqS0tpRN+Zu4MOXCXplQODFhIsW1xZyoPnHKdq2hKGc0oCjlR9Z/tZ5G09jtNPVd1dwx37ofRftQlDMaUJTyI28fepuRMSMZHz++V+43Ln4cARLQZqRXcW0xkcGRhAVpVmv1NQ0oSvmJwppCtpzYwvyU+b2WPys8KJy02DSnASUuVGsn6lQaUJTyE+8efhebsfVac1czZzPmNe2KckYDilJ+4u1DbzN6wGhGx43u1fs665jXWfLKGQ0oSvmB41XH+eLkF71eOwE4LeE04NQZ85rHSzmjAUUpP/BO7jsAzE/t/YDSumPeGENJbYnWUFQbGlCU8gNv577NhPgJjIwZ2ev3bt0xX15fTqNp1ICi2tCAopSPy6vII7Mw0yu1k2aOHfM6B0W1x1tLAF8hIrtFxCYiGQ7bU0SkRkS2W69/OOybLiKZIpIjIo+Krjuq+om3c98G4MKUC71WhgnxEyiqLeJk9UmdJa/a5ZUlgIFdwGXAk072HTDGTHWy/QlgKbAR+1LA84G3PFZCpXzEutx1TB44mWFRw7xWBsdU9o2mEdCAotrySg3FGJNljMl29XgRSQJijDGfG/tg+GeBRR4roFI+4lDZIfYW7/VqcxfA+PjxCMKe4j0U11iJIcN1lJc6lS/2oaSKyJci8pGIfMPaNgzIczgmz9qmVJ/2du7bCMIFIy/wajkigiNIjU0lqyirpclrQOgAr5ZJ+R6PNXmJyHpgiJNd9xtjXmvntGPACGNMkYhMB14VkdMAZ/0lzhe6tt97KfbmMUaMGNG1givlQ9YdWsfpg05ncORgbxeFiQkT2XRsE0mRScSGxhIU4K0Wc+WrPFZDMcacZ4xJd/JqL5hgjKkzxhRZ77cBB4Cx2GskyQ6HJgP5HVxnmTEmwxiTkZiY6J4HUqqX7S/Zz4GyAyxIXeDtogD2gFJQU8C+kn3af6Kc8qkmLxFJFJFA630aMAY4aIw5BlSIyCxrdNd1QLuBSam+4IMjHwBw3sjzvFwSu+aO+R0FOzSgKKe8NWx4sYjkAbOBtSKyztp1FrBTRHYAK4EfGGOKrX0/BP4F5GCvuegIL9Wn7SjYQVpsGgPDB3q7KIB96LAgNJkmDSjKKa80ghpjVgOrnWxfBaxq55ytQLqHi6ZUjzTaGnlo20MsHLWQcfHjun0dYww7C3Yyb/g89xWuhyKCI0iJTeFQ2SENKMopn2ryUsrfZRdn89ye57jt/dtaRkN1R15FHqV1pUxOnOzG0vVcc7OXJoZUzmhAUcqNdhTsAOyLYd390d002hq7d51C+3UmD/SxgBJvDyiadkU5owFFKTfaWbiTQeGDeHDOg2w+vpmHtz3cretkFmQSHhTOqAGj3FzCnpmUOAmAxAgdPana0oHkSrlRZkEmkxIncemoS9lVuItn9zzLxISJXJx2cZeus7NgJ6clnOZzcz2mJk7l79/8O3OGzfF2UZQP0hqKUm5SUlvCVxVftfR73D3jbqYNmsavPvsVe4v3unyduqY69pbs9bn+EwAR4ezhZxMcEOztoigfpAFFKTfJLMwEYNJAe7NQcEAwf533V2JCY7jzgzsprS116TpZRVk02hp9rv9Eqc5oQFHKTXYW7CRAAlqWzAUYGD6Qh+c9zMnqk9yz4R6abE0uXQe+7q9Qyl+4FFBEZJSIhFrv54nI7SKimeGUcrCzYCdjBowhIjjilO2TEyfz81k/5/Njn/P07qc7vU5mYSZJkUkMihjkqaIq5RGu1lBWAU0iMhr4N5AKvOCxUinlZ2zGxq7CXe32e1w25jLmDJ3Dyn0rsa/A0L6dBTtbms2U8ieuBhSbMaYRWAw8Yoz5MZDkuWIp5V9yy3KpaKjosCP9otSLOFp5tKWvxZnCmkLyq/J9skNeqc64GlAaROQa4HrgDWubDvNQyrKz0N7v0VFH+jdHfJPggOCWJX2dXsfqP9GAovyRqwHlBuyJHH9rjDkkIqnAfzxXLKX8y86CnUQHR5MSm9LuMdEh0Zw57EzW5a7DZmztXidIgpgQP8FDJVXKc1wKKMaYPcaY240xL1pfHzLG/MGzRVPKf+ws2En6wHQCpOP/UvNT5nOy+iRfnvzS6f7MwkzGxY8jLCjME8VUyqM6/OkXkUwR2dneq7cKqZQvq26oZn/pfpeaqeYNn0dYYBhvH2rb7NVka2JX4S7tkFd+q7MayreAS4C3rde11utN7OuVKNXv7S7ajc3YXAooEcERnJV8Fu8cfqdN4sgDZQeobqzW/hPltzoMKMaYw8aYw8BcY8w9xphM6/VT4MLeKaJSvq31DPnOzE+dT3FtMVtPbD1lu3bIK3/naqd8pIic2fyFiMwBIj1TJKX8y86CnYyIHuFySvdvDPsGEUERbZq9dhbsJDY0lhHRIzxRTKU8ztWAciPwuIjkisgh4P+sbd0iIleIyG4RsYlIRqt9k0Xkc2t/poiEWdunW1/niMij1trySnlV88qKXUmTEhYUxjkjzmH9V+tpsDW0bM8szGTywMnoj7byV50GFBEJAEYbY6YAk4GpxpipxpgvenDfXcBlwIZW9wrCPhz5B8aY04B5QPP/uCeApcAY6zW/B/dXyi1OVJ+goKagy4kc56fMp6yujI35GwGorK/kQOkBzd+l/FqnAcUYYwNus96XG2PKenpTY0yWMSbbya4LgJ3GmB3WcUXGmCYRSQJijDGfG3veimeBRT0th1I91bxCY1f7PeYMnUN0SHTLJMddRbswGKYMnOL2MirVW1xdveddEbkLeAmoat5ojOn+otnOjQWMiKwDEoEVxpg/AcOAPIfj8qxtSnlVZkEmIQEhjIsb16XzQgJDOHfEuaw/vJ66prqWDvn0xHRPFFO5aPOhYn79xh6CA4WEqFASIkNIiAohITKUQTGhnDt+MOEhgd4ups9yNaA095f8yGGbAdLaO0FE1gNDnOy63xjzWgflOROYAVQD74nINqDcybHtZtgTkaXYm8cYMUI7OJXn7CzcyYSECQQHdj0T0fyU+bya8yqfHv2UzIJMUmNTiQmJ8UAplSu+/KqEG57eTFxkCCkJkRwprmb7kVKKq+ppstl/3cxMiWf5jTOICPGtlTR9hUvfFWNMalcvbIw5r+vFIQ/4yBhTCCAibwLTsPerJDsclwzkd3DvZcAygIyMjI5TuyrVgZrGGk5UnXCaUqXB1sCeoj1cOe7Kbl17ZtJM4kLjePvQ2+ws3MmZw87s/KRu+t2bWcSGB/Ojc0Z77B7+bNfRMq5/ajMDo0N5+ZbZDI75OlOBzWYoq2ngvb0nuWflDm5cvoWnvzdTaypOuLzAloiki8iVInJd88sD5VkHTBaRCKuD/mxgjzHmGFAhIrOs0V3XAe3VcpRym2d3P8vC1xby7O5n26Sd31eyj7qmum7PGwkOCOa8kefx7uF3Ka4tZkqiZ/pPMvPKWLbhIH9el81LW77yyD382b4TFXz335uIDgvm+ZvOOCWYAAQECHGRISyZnsxDV05l06Fibn52K7UNnS+W1t+4usDWL4HHrNc5wJ+AS7t7UxFZLCJ52BNOrrX6TDDGlAAPAVuA7cAXxpi11mk/BP4F5AAHgLe6e3+lXFVQU4DN2Pjz1j/z4OcP0tD09TDflomIPViqd37KfBqNfca8pyY0/v2D/USHBTFnVAI/f3UXW3Ld3fXpvw4WVPLtf24iODCA5286g+S4iA6PX3T6MP68ZAqfHijklue2aVBpxdUayhLgXOC4MeYGYAoQ2t2bGmNWG2OSjTGhxpjBxpgLHfb9xxhzmjEm3Rhzj8P2rda2UcaY20xnqxQp5QYV9RUMixrGzZNuZtX+Vdyy/paWteEzCzIZGD6QpMjuLw00ffB0BoYPJDwonNED3N8cte9EBet2n+CGOSk8ce10hsdF8IPntpFXUu32e/mbI8XVXPuvTRhjeOHmM0gZ6Npc7SXTk/njZZP5aF8Btz7/BfWNbTNH19Q3sSW3mA+zT3KivLbTRdX6Cld7lmqMMTYRaRSRGOAkHXTIK9VXVNSNHFcmAAAgAElEQVRXEBsay+3Tbic1NpVffvZLrn3zWh479zF2FtpXVuzJRMTAgEB+OOWHHK86TlCA+zt6H/8gh4iQQG6Ym0psRDD/vD6DRY9/ys3PbmPVD2f3287l42W1fPtfG6mqa2TF0tmMHhTdpfOvnDGcBpuN+1fv4kcvfMG988ezM6+UL78qZfuRUrKOldNo+zqIxEeGMCEpmglDYpiQFENGShwjE/peshFXf5q2WmvI/xPYBlQCmz1WKqV8RGVDJdHB9l82l4y6hOHRw7njgzv4ztrvUNFQwaLRPZ8O1d1O/c7kFlbx+o58bvpGGnGRIQCMSozi0WtO5/vLt/D/Xt7B49+eRkBA/5qZ32Qz3P7ilxRX1vP8zbOYOLR7I+uuPWMkTTbDL17bzbt7TgAQGRLIlOEDuOXsNKYOjyM6LIi9x8rJOlbBnmPlPLvxMPWNNgIDhH9eN51vjh/szkfzOldHed1qvf2HiLyNfZKhpq9XfV5FfQUpMSktX08dNJUXL36R296/jYqSCo91pLvDEx8eICgwgJvOPHWQ5jnjBnHfggn89s0sHn1/P3eeN9ZLJfSOZRsOsjm3mL9cMYWpwwf06FrXzU5heHwEx8tqOX3EAMYMiiawVYCelZbQ8r6xycbBwir+38s7uPX5L3jh5llMG+FaDjhnjDE+larHpYAiIs8CHwMfG2P2erZISvmO8vpyokNObQ4ZGjWU5xY8x7YT28gYnNHOmd51tLSG/36ZxzUzRzAopu1iXTd9I5W9xyt4ZP1+xg2OZsGk7vcD+ZPd+WU89G42C9KHcPk098yNPmfcIJePDQoMYOzgaJ6+YQaXP/EZNy7fwsofzGH0oKgu37ehycbVyzZSWdvI/RdP4KyxiV2+hru52im/HEgCHhORAyKySkTu8FyxlPINlfWVRIW0/c8eGRzJWcln+dRfh46WfXQAY+CWs0c53S8i/HZxOlOSY3ngtd39otO4tqGJH7+0nbiIEH63uGd9Xz01MCqUZ2+cSVCAcP1TmzlRXtvlazzx4QG2HS6hpLqe657azI3Lt5BzstIDpXWdq0sAvw/8FngA+9DdDOzDeJXqsxptjVQ3Vrepofi6kxW1rNhyhMumDWPYgPB2jwsLDuSamSMorKzjYGFVu8f1FX9el82+E5X8acnklj4lbxqZEMnyG2ZSWl3P9U9tpqymofOTLFnHynns/f18a3ISH997DvctGM/mQ8XMf2QDD76+m9Lqeg+WvH2uzkN5D/gUuArIBmYYY8Z7smBKeVtlvf2vPX9Lh/Lvjw/R0GTjh/M6H4Y8IzUegC2H+vbclE9zCvn3J4e4bvZI5nWhicrT0ofF8uR3MzhQUOnyZMmGJht3vbKD2PBg/ndhOqFBgdxy9ig+vHseV84YzjOf5TLvLx+y/NNDNDa1HdLsSa42ee0E6oF07Cns00Wk/T99lOoDKhoqAIgK7nr7treUVNXzn42H+dbkoaS6MK8ibWAkA6NC2NyHJzuWVTdw1ys7SEuM5L4FE7xdnDbOHDOQv1wxhc2HirlzxfZOg8A/PjzA7vxyfrMonXiHmtbAqFB+t3gSa2//BhOTYnjm88PYerkl09VRXj8GEJEo4AbgaeyJH7s9uVEpX1dRbw8o/tTk9fRnuVTVN7mcs0tEyBgZ36dnz/9izS4KKur4761zfDb/1sKpwyioqOM3a7P49j838bdrppIU2/Zv9qxj5TxqNXXNT3c+kGJCUgzP33QGRVX1hAS5nF3LLVxt8rpNRF7Cng5lEfAUsMCTBVPK2/wtoDQ02Xjms1zOnziYcUNcL3NGShxHims4Xtb1jmFf9/KWI7y2PZ/bzx3D5OSeDRH2tJu+kcbDV01hV34ZF/3tY97LOnHK/oYmG3ev3EFMmL2pqyMiwsCo3v9739XwFY49x9Z4Y8y5xpgHrY56pfqs5j4Ufwkox0prKatp4PwJXZssN7O5H6UP1VJOltfyoxe+4J5VO5mZEs+t85yPdvM1i09P5o3/OZOk2HC+/8xWfv3GnpbULv/48AC7jrZt6vIlro7y+jMQDHwXQEQSRaTLKe2V8ifl9fZlePwloBwtrQFgaAcju5yZmBRDZEhgnwgoNpvhPxsPc+5DH/HunhP85PyxPHfTTIICe7fppyfSEqP4761zuH72SP79ySGW/OMz3tl9vKWpy5fnDLk6sfGX2IcKj8PefxKMfY2SuZ4rmlLe5W9NXs0BZVhc1wJKUGAA00bGsdnPR3plH6/gZ6sz2Xa4hNlpCfx2cTppif4zoMJRWHAgDy5MZ/aoBO5ZuZOlz20jITKEBy89zdtF65CrubwWA6cDXwAYY/JFxD/+lynVTc2jvCKD/COJ39ESe0BJim07M74zGSPjeeS9fZTVNBAb3vXVJ73t8Q9yePjdfUSHBfGXK6Zw+bRhPjvptCvmpydx2tBY/vD2Xq7KGE6CF/pFusLVgFJvjDEiYgBExD/+hynVA5X1lUQFRxEY4Jsjg1o7WlpNYnQoYcFdL++M1DiMgS8Ol3DOeN+Zp+GK17Yf5c/rsrl4chK/Xui7/QvdNTw+gse/Pc3bxXCJqw2LL4vIk8AAEbkZWI99xrxSfZazPF6+LL+0tsv9J81OHx5HcKD43XyUw0VV3L96Fxkj4/jbVVP7XDDxN67OQ/mLiJwPlGPvR/mFMeZdj5ZMKS+rqK9wmsfLVx0trWFiUvdm9YeHBJI+LNavZszXN9q4/cUvCRB45OqpftXx3le5/AkYY941xtxtjLkLeF9Eru3uTUXkChHZLSI2Eclw2H6tiGx3eNlEZKq1b7qIZIpIjog8Kn2hgVT5NMe1UHydzWY4WlrT5Q55RzNT4tmZV+Y3y9r+9Z1sduSV8aclkztdulf1jg4DiojEiMh9IvJ3EblA7G4DDgI9WRVoF3AZsMFxozHmeWPMVGPMVOxDlHONMdut3U8AS4Ex1mt+D+6vVKcq6iv8Jo9XYVUd9Y22DpNBdiYjJZ76Jhs7jpS6sWSe8WH2SZ7ccJDvzBrR7oxx1fs6q6E8h72JKxO4CXgHuAJYaIxZ2N2bGmOyjDHZnRx2DfAigIgkYV/U63NrLflnsc/YV8pj/KnJK7/UPsu9u30oABkj7Qs9bT1c4pYyecrJilruemUH44dE8/OLJ3q7OMpBZ30oacaYSQAi8i+gEBhhjKnweMnsmY2bg9YwIM9hX561TSmPqaiv8JtO+eYhwz2pocRFhjB2cBSbDxXzo3PcVTL3stkMP3lpB5V1jbx486xujWhTntNZQGlJ0G+MaRKRQ64GExFZjz2BZGv3G2Ne6+TcM4BqY8yu5k1ODms3j6aILMXePMaIESNcKa5Sp7AZm70PxV8CSmk10PVJja3NSIlnzfZ8mmymzVK2vuDJDQf5JKeQP1w2iTGD/eOz6U86CyhTRKTcei9AuPW1AMYY024DszHmvB6U62qs5i5LHpDs8HUykN/BvZcBywAyMjL6/lJ0yu2qG6qxGZvfdMofLakhOjSox5MSZ6TE8/ymr8g6Vk76sFg3lc49Pssp5K/v2OebXDVjuLeLo5zosA/FGBNojImxXtHGmCCH9x7prRSRAOz9NCscynEMqBCRWdboruuADms5SvVEZYN/JYY82oM5KI6aF9za6mPzUXJOVvKD/2wjLTGS31/m3eV7Vfu8MnBbRBaLSB4wG1grIuscdp8F5BljDrY67YfYJ1PmAAeAt3qlsKpf8sfEkD1t7gJ7H8ywAeFsyfWdjvmiyjpuWL6ZkKAAnvreDGLC/C81TH/hauoVtzLGrAZWt7PvQ2CWk+1bsa8YqZTHNSeG9JdRXkdLqltGafXUjJQ4Pj1QhDHGozWBhiYbpdUNJEa3n5+qtqGJpc9t42R5HSuWztL5Jj5Op5Yq5YQ/rSdfUdtAeW2jW5q8wD4fpaCijsNF1W65njM2m2Hps1uZ9fv3uH91Jicr2i7uZbMZ7l65k22HS3j4qqmcPsI9AVN5jgYUpZzwpyav5jko7mjygq8X3PJkXq9H3tvPB9kFzB09kJe2HOHsP33IQ+9kU1nX2HLMw+v38fqOfO6dP56LfHgNEPU1rzR5KeXrWpq8gn2/yatlyLCbaiijE6MYEBHM1txirsxw/2iq97JO8Oh7+1kyPZk/L5lMblE1f3knm0ffz+H5TV9x+7ljCAsO4LH3c7gqYzg/ODvN7WVQnqEBRSkn/GmUV/OkxmQ31VACAoSMkfEe6ZjPLazizpe2kz4sht8sSkdESB0YyePfnsbSb5Ty+7ey+OWa3QDMGZXAbxan64guP6IBRSknKuorCAsMIyTQ99OhHy2tJThQSHTj4kszUuJYn3WCoso6ty3qVF3fyC3PbSMwQHji2ultZrlPGT6AF2+exYf7Cng/6yR3XTCOYM0g7Fc0oCjlhD/l8TpaWkNSbDgBbpzZPsFKg7//ZKVbAooxhntXZbLvZAXP3DCT4fHOR2uJCOeMG8Q54/xrkS9lp+FfKSf8K49Xtdv6T5qNGmQPpjknK91yvac+zeX1HfncdcE4zhqb6JZrKt+jAUUpJ/wqoLhpUqOjobFhRIQEcqCg5wFl08EifvdmFhdMHMyt80a5oXTKV2lAUcqJivoKv8jjVd9o42RFndvmoDQTEUYlRrmlhvKXd7IZOiCMv145RTvY+zgNKEo54S+Zho+X1WIMJLs5oACMSozkYEFVj6+TW1TN7LQEojVlSp+nAUUpJ8rry/0ioOS5KW29M6MHRXG0tIYqh8mGXVXX2ERBRR3DBmjKlP5AA4pSTvjLKC93LKzVnlGJ9ufvSS3lWMtKkmFuKZPybRpQlGqlrqmOBluDX+Txak67MiTW/b+wR1sjvXrSMZ9fagU8D9SglO/RgKJUK81pV/yhU/5oaTWJ0aEeWQp3ZEIkgQHSo475vFLP1aCU79GAolQrzYkh/aLJq7TGY7+sQ4ICGBkf0eMaiggkxWpA6Q80oCjVSnPqen/olD9a4v45KI5GDerZ0OGjJTUMig4lJEh/1fQH+ikr1Upzk5ev96HYbIb8slqPNieNSowit6iKxiZbt84/Wlrj9jkyynd5awngK0Rkt4jYRCTDYXuwiDwjIpkikiUi9znsmy8i2SKSIyI/9Ua5Vf/gL6nrC6vqqG+0eTSgjB4URUOT4avi7i22le/BJjnle7xVQ9kFXAZsaLX9CiDUGDMJmA7cIiIpIhIIPA4sACYC14jIxN4ssOo/KhqsTnkfb/Ly5JDhZqN7kNPLZjPkl3q2BqV8i7fWlM8CnKVhMECkiAQB4UA9UA7MBHKMMQet81YAC4E9vVVm1X+0jPLy9YDSC0Ny0xIjATjQjbkohVV11DfZdMhwP+JrfSgrgSrgGPAV8BdjTDEwDDjicFyetU0pt6uoryBQAgkP8u1fhM1zPDzZRxETFszgmNBu1VB6owalfIvHaigish4Y4mTX/caY19o5bSbQBAwF4oCPres4yyhnOrj3UmApwIgRI7pSbKVaMg37eiLDoyU1RIcGERvu2RxZoxKjyOnG0OGjvRDwlG/xWEAxxpzXjdO+DbxtjGkATorIp0AG9tqJ4+LWyUB+B/deBiwDyMjIaDfwKOWMv6Su90TaemdGD4pi9RdHMcZ0KcjqLPn+x9eavL4Cvil2kcAsYC+wBRgjIqkiEgJcDazxYjlVH1ZRX+HzI7zAvvRvb/z1P3pQFBV1jZysqOvSec01qBjNMtxveGvY8GIRyQNmA2tFZJ2163EgCvsosC3A08aYncaYRuA2YB2QBbxsjNnthaKrfqCyodLn56CAZ1ZqdKY5SWRX+1GOltZq7aSf8dYor9XAaifbK7EPHXZ2zpvAmx4umlJU1FeQEpPi7WJ0qKK2gfLaxl5r8gJ7ksi5owe6fJ4n08Io3+RrTV5KeV15fbnP5/E62otJFwdFhxIVGtTlGkq+zpLvdzSgKNVKZb3vr9bYG0OGm4kIowZFdSlJZGVdI2U1Ddrk1c9oQFHKQaOtkerGap8PKM1zPJJ76Rf26C6uL99cPq2h9C8aUJRy0JJp2MfXQskrrSEkMIDEqNBeud+oQZGcKK+jvLbBpePzdR2UfkkDilIO/CbtSkkNSQPCCAjoncmXo7u4HLAurNU/aUBRyoG/JIbML61haC8uWtXVJJH5pTUEBwqDonunBqV8gwYUpRz4TQ2ll2bJNxsRH0FwoLjcMX+0pIak2PBeq0Ep36ABRSkH/hBQ6httnKyo69XmpKDAAFISIrtUQxk6IMzDpVK+RgOKUg78IaAcK6vBmN7PkTUqMYoDLgYU+6TGCA+XSPkaDShKOfCH1RqbO8aHx/XuL+zRg6I4XFxNfWPHywE3NNk4UV7LMK2h9DsaUJRy0Nwp78sB5eP9hYQGBXD6iAG9et/Rg6JoshkOF3U80ut4WS02L9SglPdpQFHKQWV9JVHBUQQGBHq7KO36cN9JzkhLICy4d8vYnCSys475r9PCaJNXf6MBRSkHvp7H60hxNQcLqjh7bGKv37t5OeDOOua/TgujTV79jQYUpRz4+uJaH+0rAPBKQIkMDWJobFinAUXTrvRfGlCUclDZUOnTaVc+2ldAclw4o6zaQm+zJ4nsuA/laGkNA6NCer1JTnmfBhSlHPhyDaW+0cZnOYWcPTbRa+vdj7ayDtts7a+sreug9F8aUJRy4MsBZevhYqrqm7zS3NVsVGIU1fVNHC+vbfeY3p7Fr3yHt5YAvkJEdouITUQyHLaHiMjTIpIpIjtEZJ7DvunW9hwReVS89Sea6tN8OaB8tK+A4EBhThdWTXS35pxee4+XO91vjOn1PGPKd3irhrILuAzY0Gr7zQDGmEnA+cBfRaS5jE8AS4Ex1mt+7xRV9Rc2Y6OyodJn56B8lF1Axsh4okK9snI3AFOHDyA6NIi1O4873V9cVU9tg01rKP2UVwKKMSbLGJPtZNdE4D3rmJNAKZAhIklAjDHmc2OMAZ4FFvVagVW/UN1Qjc3YiAmJ8XZR2jheVsve4xWcPc57zV0AYcGBfGtKEm/tOkZVXWOb/fml9qYwHeHVP/laH8oOYKGIBIlIKjAdGA4MA/IcjsuztinlNpUN1uJaPtjktcGLw4Vbu3xaMtX1Tby1q20t5WhpNaDroPRXHgsoIrJeRHY5eS3s4LSnsAeLrcAjwGdAI+Csv6TdYSYislREtorI1oKCgp48hupHyuvt/QK+OLHxo30FDI4JZfwQ7we76SPjSEmIYOW2I2325ZXowlr9mccaY40x53XjnEbgx81fi8hnwH6gBEh2ODQZyO/gOsuAZQAZGRntj29UyoGvZhpubLLx8f4C5qcP8dpwYUciwuXTkvnru/s4UlzN8PivU6zkl9YSERLIgIhgL5ZQeYtPNXmJSISIRFrvzwcajTF7jDHHgAoRmWWN7roOeM2bZVV9T/N68r7Wh7L9SCnltY2cPXaQt4vS4rLpyYjAf784esr2o6XVDBsQ7hOBT/U+bw0bXiwiecBsYK2IrLN2DQK+EJEs4F7guw6n/RD4F5ADHADe6sUiq36gpcnLx0Z5fbSvgACBM704XLi1YQPCmZ2WwKov8rCPk7HLL63VDvl+zCvjD40xq4HVTrbnAuPaOWcrkO7Zkqn+zFebvD7aV8C0EXHE+lgz0pLpyfzk5R1syS1hZmo8YJ/UOCk51sslU97iU01eSnmTL47yKqysY2demU+M7mptfvoQIkMCWbXNPgCzpr6J4qp67ZDvxzSgKGWpqK8gNDCUkMAQbxelxcf7reHCXp5/4kxESBALJiWxNvMYNfVNDuugaEDprzSgKGXxxbQrH2UXkBAZQvpQ32xGWjI9mcq6RtbtPv51QNFZ8v2WBhSlLL4WUGw2w4b9hZw1NpGAAN8cNTUzJZ7kuHBWfZHnsLCWBpT+SgOKUpaK+gqfWgsl82gZxVX1Ptl/0iwgwD4n5ZOcQrbmlhAYIAyODvV2sZSXaEBRyuJrNZQ3M4/ZhwuP8Z3hws5cPi0ZY+C17UcZEhNGUKD+Wumv9JNXylLZUOkzAaWitoEXNn3FgklJDIzy7b/4RyREMDM1nkab0Q75fk4DilKW8vpyn8njtWLzESrqGrnlrDRvF8UlS6bZMyNph3z/pgFFKYuvNHk1NNl46tNDzEqLZ3LyAG8XxyUXTU4iNjyYsYO9//1T3uO9lXqU8iF1TXU02Bp8Io/X6zvyOVZWy+8WT/J2UVwWFRrEh3fNIypMf6X0Z/rpK8XXaVe8ncfLGMOyDQcZOziKeT44mbEjcZG+MyFUeYc2eSnF14khvd3ktWF/IXuPV7D0rFGasVf5HQ0oSvF16npvB5RlGw4wJCaMS6cM9Wo5lOoODShK4RuZhncdLePTnCJumJtCSJD+11T+R39qlcIhoHhopnzzvJJP9he2e8yTGw4SFRrENWeM8EgZlPI07ZRXCqho8EwN5UR5LU99eogXNn5FRV0jABdPTuKX35rIoJiwluOOFFfzZuYxvn9mKjFhvrXuiVKu0oCiFO5v8tp/ooJlGw7y6vajNNkMCyYlcePcVD7LKeSxD3LYkF3APQvGc+3MEQQECP/+5BAC3DA3xS33V8obvBJQROTPwCVAPfblfG8wxpRa++4Dvg80AbcbY9ZZ2+cDfwMCgX8ZY/7gjbKrvqmivoJACSQ8qHszvY0xHCio4rMDhbyXdZKP9hUQFhzANTNHcNOZaYxIiABg+sg4vjVlKD9/NZMHXt3Fqm153Dt/PC9vPcKlU4eSFKszzZX/8lYN5V3gPmNMo4j8EbgPuFdEJgJXA6cBQ4H1IjLWOudx4HwgD9giImuMMXu8UHbVBzXPknd1qK4xhuPltXx+oIhPc4r4NKeQ4+W1gH2BqR+fN5bvzh5JvJO5GakDI/nP98/gte35/PqNPVzzz40ALPWTNCtKtcdba8q/4/DlRmCJ9X4hsMIYUwccEpEcYKa1L8cYcxBARFZYx2pAUW5RUV9BVHAUjU02TlTUcbyshmNltRwrreVEeS1FVfUUVtZRVFlPUVUdxVX1NDQZAOIigpkzaiBzRw9k7ugERsRHdBqYRIRFpw9j3rhEHnp3H6FBAYwf4v1Z+kr1hC/0odwIvGS9H4Y9wDTLs7YBHGm1/QxPFurCf85EqPPkLZQPKQqyMbghgK0PzmnZNtB6TREhOFAIDgwgKEAIDgogOD6AkCAhOiyYiJBApF7sf9508U+cAcD/Nn/xtFseRSm7IZNgQe/2DHgsoIjIemCIk133G2Nes465H2gEnm8+zcnxBufDm00H914KLAUYMaJ7QzCHEU6QrbFb5yr/M6I+kFl1IQwbEE5IUAChQQGEBAYQEhRAYIAgTn80lVKOPBZQjDHndbRfRK4HvgWca4xpDg55wHCHw5KBfOt9e9ud3XsZsAwgIyOj3cDTkadu/qg7pymlVL/llYmN1oite4FLjTHVDrvWAFeLSKiIpAJjgM3AFmCMiKSKSAj2jvs1vV1upZRS7fNWH8rfgVDgXavzcqMx5gfGmN0i8jL2luhG4EfGmCYAEbkNWId92PBTxpjd3im6UkopZ+Tr1qa+KSMjw2zdutXbxVBKKb8hItuMMRldPU9zeSmllHILDShKKaXcQgOKUkopt9CAopRSyi00oCillHKLPj/KS0QKgMPdPH0g0P6KSH2XPnf/os/dv7jy3CONMYldvXCfDyg9ISJbuzN0zt/pc/cv+tz9iyefW5u8lFJKuYUGFKWUUm6hAaVjy7xdAC/R5+5f9Ln7F489t/ahKKWUcgutoSillHILDShOiMh8EckWkRwR+am3y9NTIjJcRD4QkSwR2S0id1jb40XkXRHZb/0bZ20XEXnUev6dIjLN4VrXW8fvt9a08XkiEigiX4rIG9bXqSKyyXqGl6wlEbCWTXjJeu5NIpLicI37rO3ZInKhd57EdSIyQERWishe63Of3R8+bxH5sfUzvktEXhSRsL76eYvIUyJyUkR2OWxz22csItNFJNM651GRTta1BjDG6MvhhT09/gEgDQgBdgATvV2uHj5TEjDNeh8N7AMmAn8Cfmpt/ynwR+v9RcBb2FfQnAVssrbHAwetf+Os93Hefj4Xnv8nwAvAG9bXLwNXW+//AfzQen8r8A/r/dXAS9b7idbPQSiQav18BHr7uTp55meAm6z3IdhXG+7Tnzf25cIPAeEOn/P3+urnDZwFTAN2OWxz22eMfS2q2dY5bwELOi2Tt78pvvayvoHrHL6+D7jP2+Vy8zO+BpwPZANJ1rYkINt6/yRwjcPx2db+a4AnHbafcpwvvrCv7vke8E3gDes/RyEQ1Przxr7ezmzrfZB1nLT+GXA8zhdfQIz1i1Vabe/Tn7cVUI5YvxyDrM/7wr78eQMprQKKWz5ja99eh+2nHNfeS5u82mr+oWyWZ23rE6xq/enAJmCwMeYYgPXvIOuw9r4H/vi9eQS4B7BZXycApcaYRutrx2doeT5rf5l1vL89dxpQADxtNfX9S0Qi6eOftzHmKPAX4CvgGPbPbxt9//N25K7PeJj1vvX2DmlAactZO2GfGAonIlHAKuBOY0x5R4c62WY62O6TRORbwEljzDbHzU4ONZ3s86vnxv7X9jTgCWPM6UAV9uaP9vSJ57b6CxZib6YaCkQCC5wc2tc+b1d09Vm79T3QgNJWHjDc4etkIN9LZXEbEQnGHkyeN8b819p8QkSSrP1JwElre3vfA3/73swFLhWRXGAF9mavR4ABItK8/LXjM7Q8n7U/FijG/547D8gzxmyyvl6JPcD09c/7POCQMabAGNMA/BeYQ9//vB256zPOs9633t4hDShtbQHGWCNDQrB31q3xcpl6xBqd8W8gyxjzkMOuNUDzqI7rsfetNG+/zhoZMgsos6rP64ALRCTO+mvwAmubTzLG3GeMSTbGpGD/HN83xlwLfAAssQ5r/dzN348l1vHG2n61NSooFRiDvcPSJxljjgNHRGSctelcYA99/PPG3vhFq3QAAAM2SURBVNQ1S0QirJ/55ufu0593K275jK19FSIyy/peXudwrfZ5u1PJF1/YR0Tswz66435vl8cNz3Mm9urqTmC79boIe3vxe8B+699463gBHreePxPIcLjWjUCO9brB28/Whe/BPL4e5ZWG/RdEDvAKEGptD7O+zrH2pzmcf7/1/cjGhdEu3n4BU4Gt1mf+KvYRPH3+8wYeBPYCu4DnsI/U6pOfN/Ai9r6iBuw1iu+78zMGMqzv4wHg77Qa5OHspTPllVJKuYU2eSmllHILDShKKaXcQgOKUkopt9CAopRSyi00oCillHILDShKuUBEmkRku8OrwyzUIvIDEbnODffNFZGBPb2OUr1Bhw0r5QIRqTTGRHnhvrnY5wwU9va9leoqraEo1QNWDeKPIrLZeo22tv9KRO6y3t8uInusdShWWNviReRVa9tGEZlsbU8QkXespI5P4pBTSUS+Y91ju4g8KfZ1XgJFZLnY1//IFJEfe+HboBSgAUUpV4W3avK6ymFfuTFmJvbZxI84OfenwOnGmMnAD6xtDwJfWtt+Bjxrbf8l8ImxJ3VcA4wAEJEJwFXAXGPMVKAJuBb7jPhhxph0Y8wk4Gk3PrNSXRLU+SFKKaDG+kXuzIsO/z7sZP9O4HkReRV7GhSwp8O5HMAY875VM4nFvmjSZdb2tSJSYh1/LjAd2GItnBeOPfHf60CaiDwGrAXe6f4jKtUzWkNRqudMO++bXYw9j9J0YJuV2baj9ODOriHAM8aYqdZrnDHmV8aYEmAK8CHwI+Bf3XwGpXpMA4pSPXeVw7+fO+4QkQBguDHmA+wLfQ0AooAN2JusEJF5QKGxr1HjuH0B9qSOYE/0t0REBln74kVkpDUCLMAYswp4AHuaeqW8Qpu8lHJNuIhsd/j6bWNM89DhUBHZhP0PtGtanRcI/MdqzhLgYWNMqYj8CvuKijuBar5OOf4g8KKIfAF8hD0lO8aYPSLyc+AdK0g1YK+R1FjXaf7j8D73PbJSXaPDhpXqAR3Wq9TXtMlLKaWUW2gNRSmllFtoDUUppZRbaEBRSinlFhpQlFJKuYUGFKWUUm6hAUUppZRbaEBRSinlFv8fFNnhD2e4iXAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(aggr_ep_rewards['ep'], aggr_ep_rewards['avg'], label = 'avg')\n",
    "plt.plot(aggr_ep_rewards['ep'], aggr_ep_rewards['min'], label = 'min')\n",
    "plt.plot(aggr_ep_rewards['ep'], aggr_ep_rewards['max'], label = 'max')\n",
    "plt.legend(loc='upper left')\n",
    "plt.xlabel('Episodes')\n",
    "plt.ylabel('Rewards')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Rendering Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "done = False\n",
    "state = env.reset()\n",
    "while not done:\n",
    "    action = np.argmax(q_table[get_discrete_state(state)])\n",
    "    next_state, _, done, _ = env.step(action)\n",
    "    state = next_state\n",
    "    env.render()\n",
    "\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (spinningup)",
   "language": "python",
   "name": "spinningup"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
